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Abstract 

We consider the problem of transmission of several distributed 
sources over a multiple access channel (MAC) with side information 
at the sources and the decoder. Source-channel separation does not 
hold for this channel. Sufficient conditions are provided for transmis- 
sion of sources with a given distortion. The source and/or the channel 
could have continuous alphabets (thus Gaussian sources and Gaus- 
sian MACs are special cases). Various previous results are obtained as 
special cases. We also provide several good joint source-channel cod- 
ing schemes for a discrete/continuous source and discrete/continuous 
alphabet channel. Channels with feedback and fading are also consid- 
ered. 

Keywords: Multiple access channel, side information, lossy joint source- 
channel coding, channels with feedback, fading channels. 

1 Introduction 



In this report we consider the transmission of various sources over a multiple 
access channel (MAC). We survey the result available when the system may 
have side information at the sources and/or at the decoder. We also consider 
a MAC with feedback or when the channel experiences time varying fading. 

This system does not satisfy source-channel separation (|21j). Thus for 
optimum transmission one needs to consider joint source-channel coding. 
Thus we will provide several good joint source-channel coding schemes. Al- 
though this topic has been studied for last several decades, one recent mo- 



tivation is the problem of estimating a random field via sensor networks. 
Sensor nodes have limited computational and storage capabilities and very- 
limited energy [3j. These sensor nodes need to transmit their observations 
to a fusion center which uses this data to estimate the sensed random field. 
Since transmission is very energy intensive, it is important to minimize it. 

The proximity of the sensing nodes to each other induces high correla- 
tions between the observations of adjacent sensors. One can exploit these 
correlations to compress the transmitted data significantly. Furthermore, 
some of the nodes can be more powerful and can act as cluster heads ([6]). 
Neighboring nodes can first transmit their data to a cluster head which 
can further compress information before transmission to the fusion center. 
The transmission of data from sensor nodes to their cluster-head is usually 
through a MAC. At the fusion center the underlying physical process is esti- 
mated. The main trade-off possible is between the rates at which the sensors 
send their observations and the distortion incurred in the estimation at the 
fusion center. The availability of side information at the encoders and/or 
the decoder can reduce the rate of transmission (|82j,|31j). 

The above considerations open up new interesting problems in multi- 
user information theory and the quest for finding the optimal performance 
for various models of sources, channels and side information have made this 
an active area of research. The optimal solution is unknown except in a 
few simple cases. In this report a joint source channel coding approach is 
discussed under various assumptions on side information and distortion cri- 
teria. Sufficient conditions for transmission of discrete/continuous alphabet 
sources over a discrete/continuous alphabet MAC are given. These results 
generalize the previous results available on this problem. 

The report is organized as follows. Section [2] provides the background 
and surveys the related literature. Transmission of distributed sources over 
a MAC with side information is considered in section [3] The sources and the 
channel alphabets can be continuous or discrete. Several previous results are 
recovered as special cases in section [H Section [5] considers the important 
case of transmission of discrete correlated sources over a Gaussian MAC 
(GMAC) and presents a new coding scheme. Section [6] discusses several joint 
source-channel coding schemes for transmission of Gaussian sources over a 
GMAC and compares their performance. It also suggests coding schemes for 
general continuous sources over a GMAC. Transmission of correlated sources 
over orthogonal channels is considered in section [7J Section discusses 
a MAC with feedback. A MAC with multi path fading is addressed in 
section [9j Section [10] provides practical schemes for joint source-channel 
coding. Section [IT] gives the directions for future research and section [12] 



concludes the report. 



2 Background 

In the following we survey the related literature. Ahlswede ([T]) and Liao 
( [S] ) obtained the capacity region of a discrete memoryless MAC with inde- 
pendent inputs. Cover, El Gamal and Salehi in [21 j made further significant 
progress by providing sufficient conditions for transmitting losslessly corre- 
lated observations over a MAC. They proposed a 'correlation preserving' 
scheme for transmitting the sources. This mapping is extended to a more 
general system with several principle sources and several side information 
sources subject to cross observations at the encoders in [2]. However single 
letter characterization of the capacity region is still unknown. Indeed Duek 
[25j proved that the conditions given in [21] are only sufficient and may not 
be necessary. In |40| a single letter upper bound for the problem is obtained. 
It is also shown in [21 j that the source-channel separation does not hold in 
this case. The authors in [65J obtain a condition for separation to hold in a 
multiple access channel. 

The capacity region for distributed lossless source coding problem is 
given in the classic paper by Slepian and Wolf ([69]). Cover ([20]) extended 
Slepian-Wolf results to an arbitrary number of discrete, ergodic sources using 
a technique called 'random binning'. Other related papers on this problem 
are 0,0. 

Inspired by Slepian-Wolf results, Wyner and Ziv 82\ obtained the rate 
distortion function for source coding with side information at the decoder. 
Unlike for the lossless case, it is shown that the knowledge of the side infor- 
mation at the encoders in addition to the decoder, permits the transmission 
at a lower rate. The latter result when encoder and decoder have side 
information was first obtained by Gray and is known as conditional rate 
distortion function (See [H]). Related work on side information coding is 
[7| 1581 [24] . The lossy version of Slepian-Wolf problem is called multi-terminal 
source coding problem and despite numerous attempts (e.g., [12],[53]) the 
exact rate region is not known except for a few special cases. First major 
advancement was in Berger and Tung ([H]) where an inner and an outer 
bound on the rate distortion region was obtained. Lossy coding of contin- 
uous sources at the high resolution limit is given in [87] where an explicit 
single- letter bound is obtained. Gastpar ([32]) derived an inner and an outer 
bound with side information and proved the tightness of his bounds when 
the sources are conditionally independent given the side information. The 



authors in [72] obtain inner and outer bounds on the rate region with side 
information at the encoders and the decoder. References [71], [M] extend the 
result in [72] by requiring the encoders to communicate over a MAC, i.e., 
they obtain sufficient conditions for transmission of correlated sources over 
a MAC with given distortion constraints. In [53J achievable rate region for 
a MAC with correlated sources and feedback is given. 

The distributed Gaussian source coding problem is discussed in [54] . [76] . 
Exact rate region is provided in |76j. The capacity of a Gaussian MAC 
(GMAC) with feedback is given in [56] . In [H] a necessary and two suf- 
ficient conditions for transmitting a jointly Gaussian source over a GMAC 
are provided. It is shown that the amplify and forward scheme is optimal 
below a certain SNR determined by source correlations. The performance 
comparison of the schemes given in |44] with a Separation based scheme is 
given in [63]. GMAC under received power constraints is studied in [30] and 
it is shown that the source-channel separation holds in this case. 

In [33] the authors discuss a joint source channel coding scheme over a 
MAC and show the scaling behavior for the Gaussian channel. A Gaussian 
sensor network in distributed and collaborative setting is studied in [38] . 
The authors show that it is better to compress the local estimates than to 
compress the raw data. The scaling laws for a many-to-one data-gathering 
channel are discussed in [29|. It is shown that the transport capacity of 
the network scales as 0{logN) when the number of sensors N grows to 
infinity and the total average power remains fixed. The scaling laws for 
the problem without side information are discussed in [34] and it is shown 
that separating source coding from channel coding may require exponential 
growth, as a function of number of sensors, in communication bandwidth. 
A lower bound on best achievable distortion as a function of the number 
of sensors, total transmit power, the degrees of freedom of the underlying 
process and the spatio-temporal communication bandwidth is given. 

The joint source-channel coding problem also bears relationship to the 
CEO problem [13]. In this problem, multiple encoders observe different, 
noisy versions of a single information source and communicate it to a single 
decoder called the CEO which is required to reconstruct the source within 
a certain distortion. The Gaussian version of the CEO problem is studied 
in [55] . 

When Time Division Multiple Access (TDMA), Code Division Multiple 
Access (CDMA) or Frequency Division Multiple Access (FDMA) are used 
then a MAC becomes a system of orthogonal channels. These protocols, 
although suboptimal are frequently used in practice and hence have been 
extensively studied ([23]. [60]). Lossless transmission of correlated sources 



over orthogonal channels is addressed in [9]. The authors prove that the 
source-channel separation holds for this system. They also obtain the exact 
rate region. Reference [83] extends these results to the lossy case and shows 
that separation holds for the lossy case too. Distributed scalar quantizers 
were designed for correlated Gaussian sources and independent Gaussian 
channels in [77J. 

The information-theoretic and communication features of a fading MAC 
are given in an excellent survey paper [T3]. A survey of practical schems 
for distributed source coding for sensor networks is given in [83]. Practical 
schemes for distributed source coding are also provided in |59].|19j. 



3 Transmission of correlated sources over a MAC 

In this section we consider the transmission of memoryless dependent sources, 
through a memoryless multiple access channel (Fig. [1]). The sources and/or 
the channel input/output alphabets can be discrete or continuous. Fur- 
thermore, side information may be available at the encoders and the de- 
coder. Thus our system is very general and covers many systems stud- 
ied over the years as special cases. We consider two sources {U\,U2) and 
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Figure 1: Transmission of correlated sources over a MAC with side informa- 
tion 

side information random variables Z\, Z2, Z with a known joint distribution 
F(u\, U2, zi, Z2, z). Side information Zj is available to encoder i, i € {1,2} 
and the decoder has side information Z. The random vector sequence 
{(Ui n ,U2n, Zin, Z2 n , Z n ),n > 1} formed from the source outputs and the 
side information with distribution F is independent identically distributed 
(iid) in time. The sources transmit their codewords Aj's to a single de- 
coder through a memoryless multiple access channel. The channel output Y 
has distribution p(y\xi,X2) if x\ and X2 are transmitted at that time. The 



decoder receives Y and also has access to the side information Z. The en- 
coders at the two users do not communicate with each other except via 
the side information. It uses Y and Z to estimate the sensor observa- 
tions Ui as Ui, i £ {1,2}. It is of interest to find encoders and a de- 
coder such that {Ui n ,U 2n ,n > 1} can be transmitted over the given MAC 
with E[di(Ui,Ui)] < D\ and E[d 2 {U2, ^2)] < D2 where di are non-negative 
distortion measures and Di are the given distortion constraints. If the dis- 
tortion measures are unbounded we assume that u*, i = 1,2 exist such that 
E[di(Ui, u*)] < 00, i = 1, 2. Source channel separation does not hold in this 
case. 

For discrete sources a common distortion measure is Hamming distance 

d(x, x') = 1 if x 7^ x', 
d(x, x') = if x = x'. 

For continuous alphabet sources the most common distortion measure is 
d(x,x') = (x - x') 2 . 

We will denote {UtjJ = l,2,...,n} by U?,i = 1,2. 

Definition 1 The source (JJi^U^) can be transmitted over the multiple ac- 
cess channel with distortions D=(£>i, D2) if for any e > there is an uq 
such that for all n > hq there exist encoders /g i : lAf x Zf — > X™, i £ {1,2} 

and a decoder f%-.y n xZ n ^ {U? M2) such that \E [E"=i d(Uij, Uj) < 

Di + e, ie {1,2} where (L>f,L> 2 n ) = f p (Y n ,Z n ), U t , Z t , Z, X u y, U t are 
the sets in which Ui, Zi, Z, Xi, Y, Ui take values. 

We denote the joint distribution of (U±, U2) by p(u\, U2) and let p(y\x±, X2) 
be the transition probabilities of the MAC. Since the MAC is memory less, 
p(y n \x^,x^) = Uj=iP{yj\xij,X2j). X ^Y <-> Z will indicate that {X,Y, Z} 
form a Markov chain. 

Now we state the main Theorem. 

Theorem 1 A source can be transmitted over the multiple access channel 
with distortions (Di,!^) if there exist random variables (Wi, W2, X\, X2) 
such that 

1. p(u 1 ,U2,Z 1 ,Z 2 ,Z,Wl,W2,Xl,X 2 ,y) =p(ui,U2,zi,Z2,z)p(wi\ui,zi) 

p{w 2 \u 2 ,z 2 )p{xi\wi)p{x 2 \w 2 )p(y\xi,x 2 ) 



and 



2. there exists a function fz> : Wi x W 2 x Z — ► x W2) suc/i i/iai 
S[d(E/i,6i)] < A, * = 1,2, w/iere (f/i, f> 2 ) = /r>(Wi, W 2 , Z) and the con- 
straints 

HUuZuW^W^Z) < I(X i; Y\X 2 ,W 2 ,Z), 
I(U 2 ,Z 2 ;W 2 \W 1 ,Z) < I{X 2 -Y\X U W X ,Z), (1) 
I{U 1 ,U 2 ,Z 1 ,Z 2 ;W 1 ,W 2 \Z) < I(X 1: X 2 ;Y\Z) 

are satisfied where Wi are the sets in which Wi take values. 

In Theorem [T] the encoding scheme involves distributed quantization 
(WijWjj) of the sources {U\,U 2 ) and the side information Z\,Z 2 followed 
by correlation preserving mapping to the channel codewords (Xi,X 2 ). The 
decoding approach involves first decoding (Wi, W2) and then obtaining es- 
timate (U±,U 2 ) as a function of (Wi, W 2 ) and the decoder side information 
Z. The proof of the theorem is given in Appendix A. 

If the channel alphabets are continuous (e.g., GMAC) then in addition to 
the conditions in Theorem [1] certain power constraints E[Xf \ < P%, i = 1,2 
are also needed. 

For the discrete sources to recover the results with lossless transmission 
one can use Hamming distance as the distortion measure. 

If the source-channel separation holds then one can talk about the ca- 
pacity region of the channel. For example, when there is no side information 
Zi, Z 2 , Z and the sources are independent then we obtain the rate region 

Ri < I(X i; Y\X 2 ), R 2 < I{X 2 ;Y\X{), 

R 1 + R 2 <I(X 1 ,X 2 ;Y). (2) 

This is the well known rate region of a MAC ( [23J ) . Other special cases will 
be provided in Sec. [H 

In Theorem [T] it is possible to include other distortion constraints. For 
example, in addition to the bounds on E[d(Ui, Ui)] one may want a bound 
on the joint distortion E[d{{U\, U 2 ), {Ui, ^2))]- Then the only modification 
needed in the statement of the above theorem is to include this also as a 
condition in defining fjj. 

If we only want to estimate a function g(Ui,U 2 ) at the decoder and 
not (Ui, U 2 ) themselves, then again one can use the techniques in proof of 
Theorem[T]to obtain sufficient conditions. Depending upon g, the conditions 
needed may be weaker than those needed in ([T]) 



The main problem in using Theorem [T] is in obtaining good source- 
channel coding schemes providing (W\, W2,X\,X2) which satisfy the condi- 
tions in the theorem for a given source (U\, U2) and channel. A substantial 
part of this report will be devoted to this problem. 

3.1 Extension to multiple sources 

The above results can be generalized to the multiple (> 2) source case. Let 
5 = 1,2, M be the set of sources with joint distribution p(u%, U2---Um)- 

Theorem 2 Sources (U™,i G S) can be communicated in a distributed fash- 
ion over the memoryless multiple access channel p(y\xi,i G S) with distor- 
tions (Di,i G S) if there exist auxiliary random variables (Wi,Xi,i G S) 
satisfying 

1. p(ui, Zi, z, Wi,Xi, y,i G <S) = p(v,i, z i7 z, i G S)p(y\xi, i G S) 

Y[p(w j \u j ,z j )p(x j \wj) 
jes 

2. There exists a function fu : IljgsWj x % ~^ (^,2 G S) such that 
E[d(Ui,Ui)] < Di, i G 5 and the constraints 

I(U A ,Z A ;W A \W A o,Z) <I{X A ;Y\X A c,W A c,Z) for all A C S (3) 

are satisfied (in case of continuous channel alphabets we also need the power 
constraints E[Xf] < P i: i = l,...,M). 

3.2 Example 

We provide an example to show the reduction possible in transmission rates 
by exploiting the correlation between the sources, the side information and 
the permissible distortions. 

Consider (U\,U2) with the joint distribution: P(U\ = 0; U2 = 0) = 
P(Ui = l;U 2 = l) = l/3;P(C/i = 1;U 2 = 0) = P{U X = 0; U 2 = 1) = 1/6. 
If we use independent encoders which do not exploit the correlation among 
the sources then we need R\ > H(U\) = 1 bit and R2 > H(U2) = 1 bit 
for lossless coding of the sources. If we use the coding scheme in [69], then 
Ri > H(Ui\U 2 ) = 0.918 bits,R 2 > H(U 2 \Ui) = 0.918 bits and i?i + R 2 > 
H{Ui,U 2 ) = 1.918 bits suffice. 

Next consider a multiple access channel such that Y = X\ + X 2 where 
X\ and X2 take values from the alphabet {0, 1} and Y takes values from the 



alphabet {0, 1,2}. This does not satisfy the separation conditions (|65j). The 
sum capacity C of such a channel with independent X\ and X2 is 1.5 bits and 
if we use source-channel separation, the given sources cannot be transmitted 
losslessly because H(U\, U2) > C. Now we use a joint source-channel code 
to improve the capacity of the channel. Take X\ = U\ and X2 = C/jj. Then 
the capacity of the channel is improved to I(X\,X2]Y) = 1.585 bits. This 
is still not enough to transmit the sources over the given MAC. Next we 
exploit the side information. 

The side-information random variables are generated as follows. Z% is 
generated from U2 by using a binary symmetric channel (BSC) with cross 
over probability p = 0.3. Similarly Z2 is generated from XJ\ by using the 
same BSC. Let Z = (Z\, Z2, V), where V = U1.U2.N, N is a binary random 
variable with P(N = 0) = P(N = 1) = 0.5 independent of XJ\ and U 2 
and '.' denotes the logical AND operation. This denotes the case when the 
decoder can observe the encoder side information and also has some extra 
side information. Then from ([1]) if we use just the side information Z± the 
sum rate for the sources needs to be 1.8 bits. By symmetry the same holds if 
we only have Z^. If we use Z\ and Z2 then we can use the sum rate 1.683 bits. 
If only V is used then the sum rate needed is 1.606 bits. So far we can still 
not transmit , L^) losslessly if we use the coding Ui = Xi, i = 1,2. If 
all the information in Z\, Z2, V is used then we need R\ + R2 > 1.4120 bits. 
Thus with the aid of Z\,Z2,Z we can transmit (U\, U2) losslessly over the 
MAC even with independent X\ and X2. 

Next we consider the distortion criterion to be the Hamming distance and 
the allowable distortion as 4%. Then for compressing the individual sources 
without side information we need Ri > H (p) — H(d) = 0.758 bits, i = 1,2, 
where H(x) = —xlog2(x) — (1 — x)log2(l — x). Thus we still cannot transmit 
(1/1,1/2) with this distortion when (X\, X2) are independent. If U% and U2 
are encoded, exploiting their correlations, (X\,X2) can be correlated. Next 
assume the side information Z = (Z\,Z2) to be available at the decoder 
only. Then we need R\ > I(U%;Wi) — I{Z\\W{) where W\ is an auxiliary 
random variable generated from U\. W\ and Z\ are related by a cascade of 
a BSC with crossover probability 0.3 with a BSC with crossover probability 
0.04. This implies that R x > 0.6577 bits and R 2 > 0.6577 bits. 

4 Special Cases 

In the following we provide several systems studied in literature as special 
cases. The practically important special cases of GMAC and orthogonal 



channels will be studied in detail in later sections. There we will discuss 
several specific joint source-channel coding schemes for these and compare 
their performance. 



4.1 Lossless multiple access communication with correlated 
sources 

Take {Z\, Z 2 , Z)J-{U\, U 2 ) (X1Y denotes that r.v. X is independent of r.v. 
Y) and W\ = U\ and W 2 = U 2 where Ui, U2 are discrete sources. Then the 
constraints of (pQ) reduce to 



where X\, X2 are the channel inputs, Y is the channel output and X\ <-> 
U\ <-> U2 <-> X2 is satisfied. These are the conditions obtained in |21j . 

4.2 Lossy multiple access communication 

Take (Z±, Z2, Z)J-(Ui, U2) ■ In this case the constraints in ([I]) reduce to 



This is an immediate generalization of [21] to the lossy case. 

4.3 Lossy distributed source coding with side information 

The multiple access channel is taken as a dummy channel which reproduces 
its inputs. In this case we obtain that the sources can be coded with rates 
R\ and R2 to obtain the specified distortions at the decoder if 



H{Ui\U 2 ) < I(Xr,Y\X 2 ,U 2 ), 
H(U 2 \Ui) < I(X 2 ;Y\X 1 ,U 1 ), 
H(U U U 2 ) < I(X 1 ,X 2 ;Y) 



(4) 



I(Ui;Wi\W 2 ) < I{X 1 ;Y\X 2 ,W 2 ) 
/(C/ 2 ;W 2 |Wi) < I(X 2 ;Y\X 1 ,W 1 ) 
KUuU-2-W^Wd < I(X 1 ,X 2 ;Y). 



(5) 



R x > IiUx.Zt-Wx^Z), 
R 2 > I{U 2 ,Z 2 ;W 2 \W U Z), 
R! + R 2 > I{U U U 2 ,Z X ,Z 2 -W X ,W 2 \Z). 



(6) 



This recovers the result in [72] , and generalizes the results in [82], [31] [55] . 



4.4 Correlated sources with lossless transmission over mul- 
tiuser channels with receiver side information 

If we consider (Z\, Z 2 )-i-(Ui, U 2 ), W\ = U\ and W 2 = U 2 then we recover 
the conditions 

H(U X \U 2 ,Z) < I(Xr,Y\X 2 ,U 2 ,Z), 

H(U 2 \U 1} Z) < I(X 2 ;Y\X l ,U 1 ,Z), (7) 

H(U 1} U 2 \Z) < I{X Xl X 2 -Y\Z) 

in Theorem 2.1 in |37j . 

4.5 Mixed Side Information 

The aim is to determine the rate distortion function for transmitting a source 
X with the aid of side information (Y,Z) (system in Fig 1(c) of [27]). The 
encoder is provided with Y and the decoder has access to both Y and Z. 
This represents the Mixed side information (MSI) system which combines 
the conditional rate distortion system and the Wyner-Ziv system. This has 
the system in Fig 1(a) and (b) of [27] as special cases. The results of Fig 
1(c) can be recovered from our Theorem if we take X,Y, Z,W in [27] as 
U\ = X, Z = (Z, Y), Z\ = Y, W\ = W. U 2 and Z 2 are taken to be constants. 
The acceptable rate region is given by R > I(X,W\Y, Z), where W is a 
random variable with the property W <-> (X, Y) <-» Z and for which there 
exists a decoder function such that the distortion constraints are met. 

5 Discrete Alphabet Sources over Gaussian MAC 

This system is practically very useful. For example, in a sensor network, 
the observations sensed by the sensor nodes are discretized and then trans- 
mitted over a GMAC. The physical proximity of the sensor nodes makes 
their observations correlated. This correlation can be exploited to compress 
the transmitted data. We present a distributed 'correlation preserving' joint 
source-channel coding scheme yielding jointly Gaussian channel codewords 
which will be shown to compress the data efficiently This coding scheme 
was developed in [62j. 

Sufficient conditions for lossless transmission of two discrete sources 
(Ui,U 2 ) (generating iid sequences in time) over a general MAC with no 



side information are obtained in (J3|) and reproduced below for convenience 



H{U X \U 2 ) < I(X r ,Y\X 2 ,U 2 ) 
H{U 2 \U{) < I(X 2 ;Y\X 1 ,U 1 ) 
H(U U U 2 ) < I(X U X 2 -Y) 



(8) 



where X\ <-» U\ <-» U 2 <-» X 2 is satisfied. 

In this section, we further specialize the above results for lossless trans- 
mission of discrete correlated sources over an additive memoryless GMAC: 
Y = X\ + X 2 + N where N is a Gaussian random variable independent of X\ 
and X 2 . The noise N satisfies E[N] = and Var(N) = . We will also 
have the transmit power constraints: £?[X?] < Pi,i = 1,2. Since source- 
channel separation does not hold for this system, a joint source-channel 
coding scheme is needed for optimal performance. 

The dependence of R.H.S. of ([8]) on input alphabets prevents us from 
getting a closed form expression for the admissibility criterion. Therefore we 
relax the conditions by taking away the dependence on the input alphabets. 
This will allow us to obtain good joint source-channel codes. 

Lemma 1 Under our assumptions, I(Xi\Y\X 2 ,U 2 ) < I(X\;Y\X 2 ). 

Proof: Let 



A = H(Y\X 2 , U 2 ) - H{Y\X X ,X 2 , U 2 ) - [H(Y\X 2 ) - H(Y\X U X 2 )]. 



A = I(X r ,Y\X 2 , U 2 ) - I{X r ,Y\X 2 ). 



Then 



Since the channel is memoryless 



H{Y\X^X 2 ,U 2 )=H{Y\X X ,X 2 ). 



Thus, A = H(Y\X 2 , U 2 ) - H(Y\X 2 ) < 0. 



Therefore, from ([8]) 



H{U!\U 2 ) < I(X r ,Y\X 2 ,U 2 ) <I(X 1 ;Y\X 2 ) 
H{U 2 \U{) < I{X 2] Y\X u U l )<I{X 2 ;Y\X l ) 
H(U X ,U 2 ) < I{X U X 2 \Y). 



(9) 
(10) 
(11) 



The relaxation of the upper bounds is only in (J9j) and (|10|) and not in (jlip . 

We show that the relaxed upper bounds are maximized if (Xi,X 2 ) is 
jointly Gaussian and the correlation p between X± and X2 is high (the high- 
est possible p may not give the largest upper bound in the three inequalities 
in ©-(HU)). 

Lemma 2 A jointly Gaussian distribution for (Xi,X2) maximizes 
I{Xi;Y\X 2 ), I{X 2 ;Y\X 1 ) and I(X 1 ,X 2 ;Y) simultaneously. 

Proof: Since 

I(Xt,X 2 ;Y) = H(Y)-H(Y\X 1 ,X 2 ) 

= H(X 1 + X 2 + N)-H{N), 

it is maximized when H{X\ + X 2 + N) is maximized. This entropy is maxi- 
mized when X\+X 2 is Gaussian with the largest possible variance = P\+P 2 . 
If (Xi, X 2 ) is jointly Gaussian then so is X\ + X 2 . 
Next consider I(Xi;Y\X 2 ). This equals 

H(Y\X 2 )-H(N) = H(X 1 +X 2 + N\X 2 )- H(N) 
= H{X X + N\X 2 )-H{N) 

which is maximized when P{x\\x 2 ) is Gaussian and this happens when 
X\ , X 2 are jointly Gaussian. 

A similar result holds for I{X 2 \Y\X\). 



The difference between the bounds in Q is 

I{X 1 ,Y\X 2 )-I(X 1 ,Y\X 2 ,U 2 ) = I(X 1 + N;U 2 \X 2 ). (12) 

This difference is small if correlation between (U\, U 2 ) is small. In that case 
H(Ui\U 2 ) and H(U 2 \Ui) will be large and Q and (fTU|) can be active con- 
straints. If correlation between (U±,U 2 ) is large, H(Ui\U 2 ) and H(U 2 \Ui) 
will be small and (jlip will be the only active constraint. In this case the 
difference between the two bounds in ([9j) and (1101) is large but not impor- 
tant. Thus, the outer bounds in ([9]) and (jlpp are close to the inner bounds 
whenever the constraints ([9]) and (11Q|) are active. Often (lllj) will be the only 
active constraint. 

An advantage of outer bounds in Q and (fTUj) is that we will be able to 
obtain a good source-channel coding scheme. Once (X\,X 2 ) are obtained 



we can check for sufficient conditions (|8|). ff these are not satisfied for 
the (X\,X 2 ) obtained, we will increase the correlation p between (X±,X 2 ) 
if possible (see details below) . Increasing the correlation in [X\ , X2 ) will 
decrease the difference in (fT2j) and increase the possibility of satisfying (J5J) 
when the outer bounds in ^ and (fTUj) are satisfied. 

We evaluate the (relaxed) rate region (j9"^- (fTT|) for the Gaussian MAC 
with jointly Gaussian channel inputs (X\, X2) with the transmit power con- 
straints. For maximization of this region we need mean vector [0 0] and 

covariance matrix K v. v, = ( f-1 ^ PVP1P2 \ w k ere „ j g ^ e corre j a _ 

\P\/PiP 2 Pi J 

tion between X\ and X2. Then ([9])- (jlip provide the relaxed constraints 



H{U X \U 2 ) < 0.5 log 
H(U 2 \Ui) < 0.5 log 
H(U 1 ,U 2 ) < 0.5 log 



1 + 



1 + 



P 2 (l 



P1 + P2 + 2p^I\P 2 

(TN 2 



(13) 
(14) 
(15) 



The upper bounds in the first two inequalities in (fT3|) and (fT4"|) decrease 
as p increases. But the third upper bound (|15p increases with p and often 
the third constraint is the limiting constraint. 

This motivates us to consider the GMAC with correlated jointly Gaus- 
sian inputs. The next lemma provides an upper bound on the correlation 
between (X\,X 2 ) in terms of the distribution of (Ui, U 2 ). 

Lemma 3 Let (Ui,U 2 ) be the correlated sources and X\ U\ U 2 <-> 
X 2 where X\ and X 2 are jointly Gaussian. Then the correlation between 
(X U X 2 ) satisfies p 2 < 1 - 2- 21 ^^ . 

Proof: Since X\ *-* U\ *-* U 2 <-> X 2 is a Markov chain, by data processing 
inequality I(X\;X 2 ) < I{U\\U 2 ). Taking X\,X 2 to be jointly Gaussian 
with zero mean, unit variance and correlation p, I(Xi,X 2 ) = 0.5/og2( 1 _ L p ^ )• 
This implies p 2 < 1 - 2~ 2I{JJ ^ U2 \ 



5.1 A coding Scheme 

In this section we develop a coding scheme for mapping the discrete al- 
phabets into jointly Gaussian correlated code words which also satisfy the 



Markov condition. The heart of the scheme is to approximate a jointly Gaus- 
sian distribution with the sum of product of Gaussian marginals. Although 
this is stated in the following lemma for two dimensional vectors (Xi, A2), 
the result holds for any finite dimensional vectors. 

Lemma 4 Any jointly Gaussian two dimensional density can be uniformly 
arbitrarily closely approximated by a weighted sum of product of marginal 
Gaussian densities: 

N 

y Pi e^fri-"") 2 gj e 2^ 2 ~ a2i ) 2 , (16) 



Proof: By Stone- Weierstrass theorem (|39|) the class of functions (xi, X2) ► 
e 2c 1 ( xi ai ) e 2c 2 ( X2 a ' 2 ^ can kg shown to be dense in Co under uniform con- 
vergence where Cq is the set of all continuous functions on 3ft 2 such that 
limi|x||-»oo \ f( x )\ = • Since the jointly Gaussian density 

{x\,x%) 1— > is in Co, it can be approximated arbitrarily 

closely uniformly by the functions (|16p . 

■ 

From the above lemma we can form a sequence of functions f n (xi,X2) 
of type (fT6j) such that sup a . ljX2 |/ n (a;i, £2) — f(xi,X2)\ — » as n — ► 00, where 
/ is a given jointly Gaussian density. Although /„ are not guaranteed to 
be probability densities, due to uniform convergence, for large n, they will 
almost be. In the following lemma we will assume that we have made the 
minor modification to ensure that f n is a proper density for large enough n. 
This lemma shows that obtaining (Xi,X2) from such approximations can 
provide the (relaxed) upper bounds in (2)- (4) (we actually show for the third 
inequality only but this can be shown for the other inequalities in the same 
way). Let (X m i, X m 2) and (X\, X2) be random variables with distributions 
f m and / and sup Xl ^ 2 \f m (xi,x 2 ) - f{xi,x 2 )\ -> as m -> 00. Let Y m and 
Y denote the corresponding channel outputs. 

Lemma 5 For the random variables defined above, if {logf m (Y m ),m > 1} 
is uniformly integrable, I(X m \, X m 2]Y m ) — ► I(Xi, X 2 ;Y) osm^oo . 

Proof: Since 

I{X m i, X m 2;Y m ) = H(Y m ) — H(Y m \X m i, X m 2 
= H(Y m ) — H(N), 



it is sufficient to show that H(Y m ) — > H(Y). 

From (X m i, X m 2)— X2) and independence of (X m i, X m2 ) from iV, 
we get Y m = X ml + X m2 + iV^Xi + X 2 + N = Y . Then f m -*f uniformly 
implies that f m (Y m )-^f(Y). Since f m (Y m ) > 0, f(Y) > a.s and log is 
continuous except at 0, we obtain logf m (Y m )—^logf(Y) . Then uniform 
integrability provides I(X ml , X m2 ;Y m ) -> /(X^-X^F). 

■ 

A set of sufficient conditions for uniform integrability of {logf m (Y m ), m > 
1} is 

1. Number of components in (I16p is upper bounded. 

2. Variance of component densities in (fT6|) is upper bounded and lower 
bounded away from zero. 

3. The means of the component densities in (|16|) are in a bounded set. 

From Lemma |4] a joint Gaussian density with any correlation can be 
expressed by a linear combination of marginal Gaussian densities. But the 
coefficients pi and % in (I16p may be positive or negative. To realize our 
coding scheme, we would like to have the p^s and q^s to be non negative. 
This introduces constraints on the realizable Gaussian densities in our coding 
scheme. For example, from Lemma [3j the correlation p between X\ and X 2 
cannot exceed yl - 2~ 2I ( Ul '' U2 \ Also there is still the question of getting a 
good linear combination of marginal densities to obtain the joint density for 
a given N in (|16|) . 

This motivates us to consider an optimization procedure for finding 
Pi,qi,aii,a2i,cu and c 2 i in (fT6|) that provides the best approximation to 
a given joint Gaussian density. We illustrate this with an example. Con- 
sider U U U 2 to be binary. Let P(U X = 0; U 2 = 0) = p 00 ;P(Ui = 0; U 2 = 
1) = poi;P(Ui = 1;U 2 = 0) = pio and P{Ux = 1;U 2 = 1) = pn. We can 



consider 



f(Xi = 


■\U! 


= 0) 


= PioiAA(aioi,cioi) +Pio2AA(aio2- 

+ PlOri A/ (OlOr-i > c 10ri )> 


C102) 


(17) 


f(Xi = 


■\U! 


= 1) 


= Pm-/V(a m ,c m ) +pu2Af(au2: 

+ Pllr 2 AA(an r2 , Cn r2 ) , 


C112) 


(18) 


f(x 2 = 


■\u 2 


= 0) 


= P20iAA(a 2 oi,c 2 oi) +P202AA(a 2 o2- 

+ P20r 3 AA(a20r 3 , C20r 3 ) , 


C202) 


(19) 


f(x 2 = 


■\u 2 


= 1) 


= P2iiAA(a 2 n, c 2 ii) + P2i2AA(a 2 i 2 . 

+ P 2 lr A N{a 2 lr A -,C 2 ir A ). 


C212) 


(20) 



where J\f(a, b) denotes Gaussian density with mean a and variance b. Let 
p be the vector with components p wl , ...,p Wri ,Piii, ...,Pii r2 >P20i, -,P20r 3 > 
P 2 u, p 2 ir 4 - Similarly we denote by a and c the vectors with components 

«101) ■■■,ai0m a Hl; •■•> a llr 2 i O201, O20r 3 , C&211 , •• ■, «21r 4 an d c 101 , •■•> ClOri ) 
Cm, Curj, C 2 01, C20r 3 , C 2 n, C 2 i r - 4 . 

Let f p (xi,x 2 ) be the jointly Gaussian density that we want to approxi- 
mate. Let it has zero mean and covariance matrix Ky, Xo = I ^ f I • Let 

\p y 

9p&c be the sum of marginal densities with parameters p, a, c approximating 
f p . The best g is obtained by solving the following minimization problem: 

min p,a,c J [g E ,a,c(xi,X 2 ) - f p (xi,X 2 )] 2 dx 1 dx 2 (21) 

subject to 

ri r"2 

(POO +P0l)5I^ 1 0i a 10i + (PlO +Pll)^^ lliaili = °' 
i=l i=l 

r 3 T4 

(poo +Pio)X^ 20ia20i + + Pn)X^ 2 " a21i = °> 

i=l i=l 

ri T2 

(POO + P0l) X^ loi ( Cloi + fl 10i) + (PlO + Pll) ^Plli(cili + a lli) = 
i=l i=l 



r 3 T4 

(poo +P10) ^P 2 Qi{c 2 Qi + a| J + (poi + Pii)^P 2 ii{c 2 \i + a% u ) = 1, 

i=l i=l 



ri r 2 r 3 r 4 

Xl^ioi = h^Piii = 1 ,J2P20i = 1,^2P2U = 1, 

i=l i=l i=l i=l 



PlOi > 0,cioi > for i G {l,2...ri},pni > 0,c m > /or i G {l,2...r 2 }, 
P20i > 0,c 20 i >0 for i £ {l,2...r 3 },p 2 ii > 0,c 2 ii >0 for i £ {l,2...r 4 }. 

The above constraints are such that the resulting distribution g for (Xx, X 2 ) 
will satisfy P[Xi] = and E[Xf] = 1, i = 1, 2. 

The above coding scheme will be used to obtain a codebook as follows. 
If user 1 produces Ux = 0, then with probability pioj the encoder 1 obtains 
codeword X\ from the distribution N(axoi, cioi)- Similarly we obtain the 
codewords for U\ = 1 and for user 2. Once we have found the encoder maps 
the encoding and decoding are as described in the proof of Theorem 1 in 
Appendix A. The decoding is done by joint typicality of the received Y n 
with (C/f,C/ 2 n ). 

This coding scheme can be extended to any discrete alphabet case. We 
give an example below to illustrate the coding scheme. 

5.2 Example 

Consider (Ux, U 2 ) with the joint distribution: P(Ux = 0; U 2 = 0) = P(U\ = 
1;U 2 = 1) = P(Ux = 0;U 2 = 1) = l/3;P(E7i = 1;U 2 = 0) = and 
power constraints Pi = 3; P 2 = 4. Also consider a Gaussian multiple access 
channel with a% = 1. If the sources are mapped into independent channel 
code words, then the sum rate condition in (|15p with p = should hold. The 
LHS evaluates to 1.585 bits whereas the RHS is 1.5 bits. Thus condition 
(|15p is violated and hence the sufficient conditions in ([8]) are also violated. 

In the following we explore the possibility of using correlated (Xx, X 2 ) to 
see if we can transmit this source on the given MAC. The inputs (Ux, U 2 ) can 
be distributedly mapped to jointly Gaussian channel code words (Xx,X 2 ) 
by the technique mentioned above. The maximum p which satisfies f)13|) and 
(|14p are 0.7024 and 0.7874 respectively and the minimum p which satisfies 
(fT5j) is 0.144. Thus, we can pick a p which satisfies (fT3|) - (fT5|) . From LemmaEl 
p is upper bounded by 0.546. Therefore we want to obtain jointly Gaussian 
(Xx,X 2 ) satisfying Xx Ux <->• U 2 <-> X 2 with correlation p G [0.144,0.546]. 
If we pick a p that satisfies the original bounds, then we will be able to 
transmit the sources (Ux,U 2 ) reliably on this MAC. Without loss of gen- 
erality the jointly Gaussian channel inputs required are chosen with mean 

vector [0 0] and covariance matrix Kx x ,x 2 = \ \\- The p chosen is 0.3 



and hence is such that it meets all the conditions ()13[) - (jl5[) . Also, we choose 
r\ = r 2 = ^3 = r4 = 2. We solve the optimization problem (|2ip via MAT- 
LAB to get the function g. The normalized minimum distortion, defined as 
/ [dp,a,c{xi,x 2 ) - f P (xi,X2)] 2 dxidx2/ J fp (x\ , x 2 )dx\dx 2 is 0.137% when the 
marginals are chosen as: 

/(Xtpt = 0) = AA(-0002, 0.9108), /(Xl|E7i = 1) = AA(-0001, 1.0446), 
f(X 2 \U 2 = 0) = AA(-0021, 1.1358), f(X 2 \U 2 = 1) = AA(-0042, 0.7283). 

The approximation (a cross section of the two dimensional densities) is 
shown in Fig. 15. 2[ 

If we take p = 0.6 which violates Lemma [3] then the approximation is 
shown in Fig. 15. 2[ We can see from Fig. 15.21 that the error in this case is 
more. Now the normalized marginal distortion is 10.5 %. 

The original upper bound in ([9]) and (|10|) for this example with p = 0.3 
is I(X X -Y\X 2 ,U 2 ) = 0.792, I(X 2 ;Y\Xx, Ui) = 0.996. Also, I(Xi;Y\X 2 ) = 
0.949, I(X 2 ;Y\Xi) = 1.107. H(U!\U 2 ) = fl"({7 2 |^l) = 0.66 and we conclude 
that the original bounds too are satisfied by the choice of p = 0.3. 
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Figure 2: Cross section of the approximation of the joint Gaussian p=0.3 



6 Source- Channel Coding for Gaussian sources over 
Gaussian MAC 

In this section we consider transmission of correlated Gaussian sources over a 
GMAC. This is an important example for transmitting continuous alphabet 
sources over a GMAC. For example one comes across it if a sensor network 




Figure 3: Cross section of the approximation of the joint Gaussian p=0.6 



is sampling a Gaussian random field. Also, in the application of detection 
of change (|74|) by a sensor network, it is often the detection of change in 
the mean of the sensor observations with the sensor observation noise being 
Gaussian. 

We will assume that (Ui n , U 2n ) is jointly Gaussian with mean zero, vari- 
ances af, i = 1,2 and correlation p. The distortion measure will be Mean 
Square Error (MSE). The (relaxed) sufficient conditions from © for trans- 
mission of the sources over the channel are given by (these continue to hold 
because Lemmas QJ3] are still valid) 



7(t7i;Wi|W 2 ) < 0.5 log 
I(U 2 ;W 2 \W 1 ) < 0.5 log 
/([/i,C/ 2 ;Tyi,W 2 ) < 0.5 log 



1 + 



Pi(l-P 2 ) 



1 P2(1~P 2 ) 



1 + 



ON 

P!+P 2 + Ip^fP^l 



(22) 



We consider three specific coding schemes to obtain W\, W 2 , Xi, X 2 where 
(Wx, W 2 ) satisfy the distortion constraints and (X\,X 2 ) are jointly Gaussian 
with an appropriate p such that (|22p is satisfied. These coding schemes have 
been widely used. We compare their performance also. 



6.1 Amplify and forward scheme 

In the Amplify and Forward (AF) scheme the channel codes X% are just 
scaled source symbols t/j. Since (Ui,U 2 ) are themselves jointly Gaussian, 



{X\,X 2 ) will be jointly Gaussian and retain the dependence of inputs (U±, U2) 
The scaling is done to ensure -EpQ 2 ] = Pi,i = 1,2. For a single user case 
this coding is optimal [23] . 

At the decoder inputs U% and U2 are directly estimated from Y as Ui = 
E[Ui\Y], i = 1,2. Because Ui and Y are jointly Gaussian this estimate 
is linear and also satisfies the Minimum Mean Square Error (MMSE) and 
Maximum Likelihood (ML) criteria. 

The MMSE distortion for this encoding-decoding scheme is 

_ < Ti 2 [w-P 2 )+^ 2 ] w ^[m-ft+vN 2 ] (23) 

1 P 1 + P2 + 2p^fI\P2 + (TN 2, 2 Pi+P 2 + 2pyfl\P2~ + a N 2 ' 1 ' 

Since encoding and decoding require minimum processing and delay in 
this scheme, if it satisfies the required distortion bounds Di, it should be the 
scheme to implement. This scheme has been studied in |44j and found to be 
optimal below a certain SNR for two- user symmetric case (Pi = Pi^x = 
o"2,.Di = D2). However unlike for single user case, in this case user 1 acts 
as interference for user 2 (and vice versa). Thus one should not expect this 
scheme to be optimal under high SNR case. That this is indeed true was 
shown in Ref.[63j. It was also shown there, that at high SNR, for P\ ^ P2, 
it may indeed be better in AF to use less power than P\,P2. This can also 
be interpreted as using AF on U\ — a.\E\U2\U{\ and U2 — oe2E[Ui\U2] at the 
two encoders at high SNR which will reduce the correlations between the 
transmitted symbols. 

6.2 Separation based scheme 
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Figure 4: separation based scheme 

In separation based (SB) approach (Fig. H]) the jointly Gaussian sources 
are vector quantized to and ■ The quantized outputs are Slepian- 
Wolf encoded [69]. This produces code words, which are (asymptotically) 



independent. These independent code words are encoded to capacity achiev- 
ing Gaussian channel codes (X^X^) with correlation p = 0. This is a very 
natural scheme and has been considered by various authors (|21j,|65j,|23j). 

Since source-channel separation does not hold for this system, this scheme 
is not expected to be optimal. But because this scheme decouples source 
coding from channel coding, it is preferable to a joint source-channel coding 
scheme with comparable performance. 



6.3 Lapidoth-Tinguely scheme 

In this scheme, obtained in [H], (C/f , UQ) are vector quantized to 2 1 , 2 nR2 
01, U2) vectors where Ri and R2 will be specified below. Also, Wf, 
are 2 nRl and 2 nR% , n length code words obtained independently with distri- 
butions Af(0, 1). For each uf, we pick the codeword wf that is closest to it. 
This way we obtain Gaussian codewords W™, W£ which retain the correla- 
tions of (Uf , U^)- Xf and X% are obtained by scaling Wi, W£ to satisfy the 
transmit power constraints. We will call this LT scheme. (Ui,U2,Wi,W2) 
are (approximately) jointly Gaussian with covariance matrix 



07 



pa ^2 

p<J\(J2 of po"icr 2 (l - 2~ 2Rl ) 

ff 2( 1 _ 2 -2fli) pa 1 a 2 (l-2- 2R ^) a\{\ - 2~ 2iil ) 

\pa l a 2 {l-2-^) al{\-2-^) 



a 2 i{l _ 2 -2 Rl) p(Tl(T2 (l-2-* R z)\ 



a 2 (l 



2 -2iJ 2 ) 



f(l-2- 2 *) ) 
(24) 



In dMD P = P\]{1 ~ 2- 2iJ i)(l - 2- 2 «2). 

We obtain the -R2) above from ([22]) . From 

I{U 1 ;Wx\W 2 )=H{W 1 \W 2 )-H{W 1 \W2,U 1 ), 

and the fact that the Markov chain condition W\ «-> C/'i <-> C/2 VF2 holds, 

ff(Wi|W 2j l7i) = JT(Wi|C7i) 

and 

J(^i;Wi|T7 2 ) = 0.51og [(l-p 2 )2 2iJl 
Thus from (|22p we need i?i and -R2 which satisfy 



ill < 0.5 log 



+ 



P 2 ) 



(25) 



Similarly, we also need 



R 2 < 0.5 log 
R1 + R2 < 0.5 log 



0"AT 



+ 



(1-P 2 



qjy 2 + Pi + P 2 + 2~pJT\P 2 

(1 " P 2 )^ 2 



(26) 
(27) 



The inequalities (|25p -(|27p are the same as in |44| . Thus we recover the 
conditions in [44] from our general result (([I])). Taking U{ = E[U~i\Wi, W2], i = 
1,2, we obtain the distortions 



Di = var(U 1 \W 1 ,W 2 ) 



D 2 = var{U 2 \W u W 2 ) 



(1 - f?) 

a 2 2 2- 2R * [l - p 2 (l - 2- 2 ^ 



(28) 



(29) 



(1 - p 2 ) 

The minimum distortion is obtained when p is such that the sum rate is 
met with equality in (|27p . For the symmetric case at the minimum distor- 
tion, R\ = R 2 . 

6.4 Asymptotic performance of the three schemes 

We compare the performance of the three schemes. These results are from 
|63j . For simplicity we consider the symmetric case: P\ = P2 = P, o\ = 
a 2 = a, D 1 = D 2 = D. We will denote the SNR P/a N 2 by S. 
Consider the AF scheme. From (12311 



D(S) 



a 2 [S (1 - p 2 ) + 1] 



(30) 



2S{l + p) + l 

Thus D(S) decreases to cr 2 (l — p)/2 strictly monotonically at rate O(l) as 
S — > 00. 
Also, 

D(S) - a 2 



lim 

5-»0 



S 



a\l + pf. 



(31) 



Thus, D(S) -> a 2 at rate 0(5) as S -» 



Consider the SB scheme at High SNR. From [76] if each source is encoded 
with rate R then it can be decoded at the decoder with distortion 



2- AR (l-p 2 )+p 2 2 



2 -8R 



(32) 



At high SNR, from the capacity result for independent inputs, we have 
R < 0.25 log S ([23]). Then from ([32} we obtain 



and this lower bound is achievable. As S — > do, this lower bound approaches 
zero at rate 0(yS). Thus this scheme outperforms AF at high SNR. 
At low SNR, R w | and hence from (gg) 

D>p 2 a 4 2- iS + a 2 (l-p 2 )2- 2S . (34) 

Thus D — > <7 2 at rate 0(S 2 ) as £ — > at high p and at rate 0(5) at small p. 
Therefore we expect that at low SNR, at high p this scheme will be worse 
than AF but at low p it will be comparable. 

Consider the LT scheme. In the high SNR region we assume that 
p = p since R = R\ = R2 are sufficiently large. Then from (|27[) R ~ 
0.25log[2S/(l — p)} and the distortion can be approximated by 



D*a^(l-p)/2S. (35) 

Therefore, D — > as S — > 00 at rate 0(y/S). This rate of convergence is 
same as for SB. However, the R.H.S. in (|33h is greater than that of (135j) and 
at low /? the two are close. Thus at high SNR LT always outperforms SB 
but the improvement is small for low p. 

At low SNR 

S(l + p) log(l - p 2 ) 

R 2 1 

and evaluating D from ([28]) we get 

a 2 2~s (l- p 2 (l- ^TTj22- : 
D = i 7 == ^ (36) 

where S = 5(1 + /5). Therefore D -> a 2 as S 1 at rate 0(S 2 ) at high 
p and at rate 0(5) at low p. These rates are same as that for SB. In fact, 
dividing the expression for D at low SNR for SB by that for LT, we can 
show that the two distortions tend to a 2 at the same rate for all p . 

The necessary conditions (NC) to be able to transmit on the GMAC 
with distortion (D,D) for the symmetric case are (|44j) 



f **[s(i-(?)+: 

2S(l+p)+l 



D > I 



for S<j^, 



(37) 



a 



2 1 { -^~ lor.S -j^ 



25(1+70+1 



The above three schemes along with (|37p are compared below using exact 
computations. Figures [5] and [6] shows the distortion as a function of SNR for 
unit variance jointly Gaussian sources with correlations p = 0.1 and 0.75. 



SNR vs Distortion performance 
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Figure 5: SNR vs distortion performance p=0.1 



SNR vs Distortion performance 
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Figure 6: SNR vs distortion performance p=0.75 
From these plots we confirm our theoretical conclusions provided above. 



6.5 Continuous sources over a GMAC 

For general continuous alphabet sources (Ui, U 2 ) we vector quantize £7f , 
into Ui,U 2 . Then to obtain correlated Gaussian codewords (X¥,X%) we 
can use two schemes adapted from the cases studied above. In the first 
scheme we use the scheme developed in sec. 15.11 In the second scheme we 
use LT scheme explained in sec. 16.31 

7 Correlated Sources over Orthogonal MAC 

One standard way to use the MAC is via TDM A, FDMA, CDMA or Orthog- 
onal Frequency Division Multiple Access (OFDMA) ( [231 EDI M ) • These pro- 
tocols although suboptimal are used due to practical considerations. These 
protocols make a MAC a set of parallel orthogonal channels (for CDMA, it 
happens if we use orthogonal codes). We study transmission of correlated 
sources through such a system. 

7.1 Transmission of correlated sources over orthogonal chan- 
nels 

Consider the setup in Fig.[T]when Y = (Y]_, Y 2 ) and p(y\x\, X2) = p(yi,y2\xi, X2) 
p(yi\xi)p(y2\x2) ■ Then the conditions in (JTJ) become 

HjJ u Z^Wx\W 2 ,Z) < I(X 1 ;Y 1 \W 2 ,Z)<I(X 1 ;Y 1 ), (38) 
I(U2,Z 2 ;W 2 \W 1 ,Z) < I(X2;Y 2 \W 1 ,Z)<I(X 2 ;Y2), (39) 
I(U 1 ,U2,Z 1 ,Z 2 ;W 1 ,W 2 \Z) < I(X l ,X 2 ;Y 1 ,Y 2 \Z)<I(X l ;Y 1 ) + I(X 2 ;Y 2 ). 

(40) 

The outer bounds in (J38l) - (l40l) are attained if the channel codewords (Xi, X2) 
are independent of each other. Also, the distribution of (X\,X 2 ) maximiz- 
ing these bounds are not dependent on the distribution of {JJ\iU 2 )- This 
implies that source-channel separation holds for this system even with side 
information (Z\, Z 2 , Z) (for the sufficient conditions ([I])). Thus by choosing 
(Xi,X 2 ) which maximize the outer bounds in ([55)1 - (|40 p we obtain capacity 
region for this system which is independent of the side conditions. Also, for 
a GMAC this is obtained by independent Gaussian r.v.s X\ and X 2 with 
distributions JV(0, Pj), % = 1,2, where Pj are the power constraints. Fur- 
thermore, the L.H.S. of the inequalities are simultaneously minimized when 



W\ and W 2 are independent. Thus, the source coding (Wi, W2) on (Ui, Z\) 
and can be done as in Slepian-Wolf coding (by first vector quan- 

tizing in case of continuous valued Ui,U 2 ) but also taking into account the 
fact that the side information Z is available at the decoder. In this section 
this coding scheme will be called SB. 

If we take W\ = XJ\ and W 2 = U2 and the side information (Z\, Z 2 , Z) 
_L (Ui, U2), we can recover the conditions in [9]. 

7.2 Gaussian sources and orthogonal Gaussian channels 

Now we consider the transmission of jointly Gaussian sources over orthogo- 
nal Gaussian channels. Initially it will also be assumed that there is no side 
information [Z\, Z 2 , Z). 

Now (Ui, U2) are zero mean jointly Gaussian random variables with vari- 
ances erf and a 2 respectively and correlation p. Then = JQ + iVj, i = 1,2 
where N{ is Gaussian with zero mean and aj^. variance. Also N\ and N2 are 
independent of each other and also of (Ui,U 2 ). 

In this scenario, the R.H.S. of the inequalities in (|38f )- (|40h are maximized 
by taking Xi ~ A^(0, Pi), i = 1, 2 independent of each other where Pi is the 
average transmit power constraint on user i. Then I(Xi,Yi) = 0.5log(l + 



Based on the comments at the end of sec. 17. H for two users, using the 
results from [76] we obtain the necessary and sufficient conditions for trans- 
mission on an orthogonal GMAC with given distortions D\ and D2. 

We can specialize the above results to a TDMA, FDMA or CDMA based 
transmission scheme. The specialization to TDMA is given here. Suppose 
source 1 uses the channel a fraction of time and user 2, 1 — a fraction of time. 
In this case we can use average power P\ja for the first user and P2AI — a ) 
for the second user whenever they transmit. The conditions (I38p - (]40p for 
the optimal scheme become 



= 1,2. 



J(E/i;Wi|W 2 ) < 0.5a log 1 + 



p i 1 



(41) 



I(U 2 ;W 2 \Wi) < 0.5(1 -a) log 1 + 



P2 



(42) 



(1 - a)a N2 



I(Ui,U 2 ;Wi,W 2 ) < 0.5a log 1 + 



Pi 1 



a<7/Vi 2 



+ 0.5(1 -a) log 1 + 



Pi 



(43) 



(1 - a)<JN 2 



In the following we compare the performance of the AF scheme (ex- 
plained in sec. 16, ip with the SB scheme. Unlike in the GMAC there is 
no interference between the two users when orthogonal channels are used. 
Therefore, in this case we expect AF to perform quite well. 

For AF, the minimum distortions (Di,D 2 ) are 



Do 



P 2 (l-p 2 ) + ( J 2 



A ? 2 



PiP 2 (l ~ P 2 ) + o-lPx + a 2 P 2 + a 



'N 1 cr N 2 



<T2VN 2 ) 2 Pi(l-p 2 ) + a 



Ni 



PM1 



+ a A N2 P 1 + a I Ni P 2 + a 2 Ni u N2 



- 2 rr 2 



(44) 
(45) 



Thus, as P%, P 2 — ► oo, D\, D 2 tend to zero. We also see that D\ and D 2 are 
minimum when the average powers used are Pi and P 2 . These conclusions 
are in contrast to the case of a GMAC where the distortion for the AF does 
not approach zero as P\,P 2 — > oo and the optimal powers needed may not 
be the maximum average allowed Pi and P 2 (|63j). 

We compare the performance of AF with SB for the symmetric case 
where Pi = P 2 = P, a 2 = a\ = a 2 , D 1 = D 2 = D, a 2 Ni = a% 2 = a 2 N . These 
results are from |61j . 

We denote the minimum distortions achieved in SB and AF by D(SB) 
and D(AF) respectively, a 2 is taken to be unity without loss of generality. 
We denote P/a 2 N by S. Then 



D{SB) = VOW + W ( } = l + 2S + S 2 (l - p 2 ) ■ (46) 

We see from the above equations that when p = 0, D(SB) = D(AF) = 
1/(1 + S). At high S, D(AF) « l/S and D(SB) w ^T^f /S. Eventually 
both D(SB) and D(AF) tend to zero as S -> oo. When S -> both D(SB) 
and D(AF) go to a 2 . 

By squaring the equation (|46p we can show that D(AF) > D(SB) for 
all S. But in [6lj we have shown that D(AF) — D(SB) is small when S is 
small or large or whenever p is small. 

D(AF) and D(SB) are plotted for p=0.3 and 0.7 using exact computa- 
tions in Figs. [7] and [HJ 

The above results can be easily extended to the multiple source case. For 
SB, for the source coding part, the rate region for multiple user case (under 
a symmetry assumption) is given in [75J. This can be combined with the 
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Figure 8: SNR vs distortion performance rho=0.7 



capacity achieving Gaussian channel codes over each independent channel 
to obtain the necessary and sufficient conditions for transmission. 

Let N be the number of sources which are jointly Gaussian with zero 
mean and covariance matrix K\j . Let P be the symmetric power constraint. 

Let Kjj have the same structure as given in [75]. Let Cjjy = y/P[lp p] 

be a 1 x N vector. The minimum distortion achieved by the AF scheme is 
given as D(AF) = 1 - C UY (PKu + a 2 N I)- l C' UY . 



7.3 Side information 

Let us consider the case when side information Zi is available at encoder i, 
i = 1, 2 and Z is available at the decoder. One use of the side information Zj 
at the encoders is to increase the correlation between the sources. This can 



be optimally done (see [IS]), if we take appropriate linear combination of 
(Ui, Zi) at encoder i. The following results are from [61]. We are not aware 
of any other result on performance of joint source-channel schemes with side 
information. We are currently working on obtaining similar results for the 
general MAC. 

7.3.1 AF with side information 

Side information at encoders only: A linear combination of the source 
outputs and side information Li = a^Ui + biZi, % = 1, 2 is amplified and sent 
over the channel. We find the linear combinations, which minimize the sum 
of distortions. For this we consider the following optimization problem: 
Minimize 

£>(ai, 61,03, 62) = E[(U X - U x ) 2 } + E[(U 2 - U 2 ) 2 ) (47) 

subject to 

E[X 2 } < Pi, E[X 2 } < P 2 

where 

U x = E[U X \Y X ,Y 2 ], U 2 = E[U 2 \Y X ,Y 2 ], 

Side information at Decoder only: In this case the decoder side 
information Z is used in estimating (U X ,U 2 ) from (Y X ,Y 2 ). The optimal 
estimation rule is 

U x = E[U X \Y X , Y 2 , Z],U 2 = E[U 2 \Yx,Y 2 , Z\. (48) 

Side information at both Encoder and Decoder: Linear combina- 
tions of the sources are amplified as above and sent over the channel. To 
find the optimal linear combination, solve an optimization problem similar 
to flUD with (U X ,U 2 ) as given in (g8]). 

7.3.2 SB with side information 

For a given (L X ,L 2 ) we use the source-channel coding scheme explained at 
the end of sec. 17.11 The side information Z at the decoder reduces the source 
rate region. This is also used at the decoder in estimating (U X ,U 2 ). The 
linear combinations L x and L 2 are obtained which minimize (|47p through 
this coding-decoding scheme. 



7.3.3 Comparison of AF and SB with side information 



We provide the comparison of AF with SB for U±, U 2 ~ A/"(0, 1). Also we 
take the side information with a specific structure which seems natural in 
this set up. Let Z x = s x U 2 + V\ and Z 2 = s 2 Ui + V 2 , where Vi, V 2 ~ M(0, 1) 
and are independent of each other and independent of the sources, and s± 
and s 2 are constants that can be interpreted as the side channel SNR. We 
also take Z = (Z±, Z 2 ). 

We have compared AF and SB with different p and s%,s 2 by explicitly 
computing the minimum {D\ + D 2 )/2 achievable. We take Pi = P 2 . For 
s\ = s 2 = 0.5 and p = 0.4 we provide the results in Fig. [H From the Figure 
one sees that without side information, the performance of AF and SB is 
very close for different SNRs. The difference in their performance increases 
with side information for moderate values of SNR because the effect of the 
side information is to effectively increase the correlation between the sources. 
Even for these cases at low and high SNRs the performance of AF is close 
to that of SB. These observations are in conformity with our conclusions in 
the previous Section . 

Our other conclusions, based on computations not provided here are the 
following. For the symmetric case, for SB, encoder-only side information 
reduces the distortion marginally. This happens because a distortion is 
incurred for (Ui, U 2 ) while making the linear combinations (Li, L 2 ). For the 
AF we actually see no improvement and the optimal linear combination has 
b\ = b 2 = 0. For decoder-only side information the performance is improved 
for both AF and SB as the side information can be used to obtain better 
estimates of (U\, U 2 ). Adding encoder side information further improves the 
performance only marginally for SB; the AF performance is not improved. 

In the asymmetric case some of these conclusions may not be valid. 



In this section we consider a memoryless MAC with feedback. The channel 
output Yk-i is available to the encoders at time k. 

Gaarder and Wolf (|28j) showed that, unlike in the point to point case, 
feedback increases the capacity region of a discrete memoryless multiple- 
access channel . In |22| an achievable region 




8 MAC with feedback 




Pi < I(X 1 ;Y\X 2 , U), R 2 < I(X 2 ; Y\X U U) 
R 1 + R 2 <I(X 1 ,X 2 ;Y) 
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Figure 9: AF and SB with both encoder and decoder side information 

where p(u, x±, X2,y) = p(u)p(xi\u)p(x2\u)p(y\xi, X2). It was demonstrated 
in |78j that the same rate region is achievable if there is a feedback link to 
only one of the transmitters. This achievable region was improved in [IS]. 

The achievable region for a MAC, where each node receives possibly 
different channel feedback, is derived in [T7j. The feedback signal in their 
set-up is correlated but not identical to the signal observed by the receiver. 
A simpler and larger rate region for the same set-up was obtained in |79j . 

Kramer ([43]) used the notion of 'directed information' to derive an ex- 
pression for the capacity region of the MAC with feedback. However, no 
single letter expressions were obtained. 

If the users generate independent sequences, then the capacity region 
Cfb of the white Gaussian MAC is ([56]) 



The capacity region for a given p in (|50p is same as in (|13|) - ()15|) for a channel 
without feedback but with correlation p between channel inputs (Xi, X2). 
Thus the effect of feedback is to allow arbitrary correlation in {X\,Xi)- 

An achievable region for a GMAC with noisy feedback is provided in 
[46] . Gaussian MAC with different feedback to different nodes is considered 




(50) 



Ri + R 2 < 0.5 log 1 H k 



} 



in [66]. An achievable region based on cooperation among the sources is also 
given. 

Reference jSDJ obtains an achievable region when non-causal state infor- 
mation is available at both encoders. The authors also provide the capacity 
region for a Gaussian MAC with additive interference and feedback. It is 
found that feedback of the output enhances the capacity of the MAC with 
state. Interference when causally known at the transmitters can be exactly 
cancelled and hence has no impact on the capacity region of a two user 
MAC. Thus the capacity region is the same as given in (|50p . 

In [50], it is shown that feedback does not increase the capacity of the 
Gelfand-Pinsker channel ( [35] ) and feedforward does not improve the achiev- 
able rate-distortion performance in the Wyner-Ziv system (|82j). 

MAC with feedback and correlated sources (MACFCS) is studied in 
[53\ [52] . This has a MAC with correlated sources and a MAC with feedback 
as special cases. Gaussian MACFCS with a total average power constraint 
is considered in [52]. Different achievable rate regions and a capacity outer 
bound are given for the MACFCS in [53J. For the first achievable region a 
decode and forward based strategy is used where the sources first exchange 
their data, and then cooperate to send the full information to the destina- 
tion. For two other achievable regions, Slepian-Wolf coding is performed 
first to remove the correlations among the source data and it is followed by 
the coding for the MAC with feedback or MAC disregarding the feedback. 
The authors also show that different coding stategies perform better under 
different source correlation structures. 

The transmission of bivariate Gaussian sources over a Gaussian MAC 
with feedback is analyzed in [45]. The authors show that for the symmetric 
case, for SNR less than a threshold which is determined by the source cor- 
relation, feedback is useless and minimum distortion is achieved by uncoded 
transmission. 

9 MAC with fading 

A Gaussian MAC with a finite number of fading states is considered. We 
provide results when there are M independent sources. The channel state 
information (CSI) may be available at the receiver and/or the transmitters. 
Consider the channel ([14]) 



M 

Y k = Y, h lk x lk + N k 
i=i 



(51) 



where Xik is the channel input and hik is the fading value at time k for user 
I. The fading processes {hik, k > 1} of all users are jointly stationary and 
ergodic and the stationary distribution has a continuous bounded density. 
The fading process for the different users are independent. {N^} is the 
additive white Gaussian noise. All the users are power constrainted to P, 
i.e., E[Xf] < P for all I. 

Since the source-channel separation holds, we provide the capacity region 
of this channel. 



9.1 CSI at receiver only 



When the channel fading process {hik} is available at the receiver only, the 
achievable rate region is the set of rates (i?i,i?2, —,Rm) satisfying 
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log[l + 



(52) 



for all subsets S of {1, 2....M}, v x = \hi\ 2 and a 2 = E[N 2 ]. The expectation 
is over all fading powers {vi}, I € S. One of the performance measures is 
normalized sum rate per user 
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(53) 



If E[v{\ = 1 for each i, then the upper bound equals the capacity of the 



AWGN channel ±log 1 + ^£ . Also, as M increases, if {v{\ are iid, by 
Law of Large Numbers (LLN), R will be close to this upper bound. Thus 
averaging over many users mitigates the effect of fading. This is in contrast 
to the time /frequency /space averaging. 

The capacity achieving distribution is iid Gaussian for each user and the 
code for one user is independent of the code for another user (in other words 
AF is optimal in this case). 



9.2 CSI at both Transmitter and Receiver 

The additional element that is introduced when CSI is provided to the trans- 
mitters in addition to the receiver is dynamic power control which can be 
done in response to the changing channel state. 



Given a joint fading power v = (fx, I'm), Pii^) denotes the transmit 
power allocated to user i. Let Pi be the average power constraint for user i. 
For a given power control policy V 

1, (-, , T,ies "iPiM 



C f (V) = < R : R(5) < £ 



( 1 + ' J for all S C {1, 2, M}| 



2 J V ^ : 

(54) 

denotes the rate region achievable. The capacity region is 

C(P) = (J C f (V) (55) 

where T is the set of feasible power control policies, 

T = {V : E[Pi(v)\ < P h for i = 1, M}. (56) 

Since the capacity region is convex, the above characterization implies that 
time sharing is not required. 

The explicit characterization of the capacity region exploiting its poly- 
matroid structure is given in [70] . For Pj = P for each i and each hi having 
the same distribution, the optimal power control is that only the user with 
the best channel transmits at a time. The instantaneous power assigned to 
the ith user, observing the realization of the fading powers v\,v<i,..., I'm is 

It — Vi > A, Vi > za- 7 ^ I 
Ptivj, j = l,...,M) = {* v u l . 3 JT (57) 

\\J otherwise 

where A is chosen such that the average power constraint is satisfied. This 
function is actually the well known water filling function (|36j) optimal for 
a single user. This strategy does not depend on the fading statistics but for 
the constant A. The capacity achieving distribution is Gaussian (thus AF 
for each user in its assigned slot is optimal). 

Unlike in the single user case the optimal power control may yield sub- 
stantial gain in capacity. This happens because if M is large, with high 
probability at least one of the iid fading powers will be large providing a 
good channel for the respective user at that time instant. 

The optimal strategy is also valid for non equal average powers. The only 
change being that the fading values are normalized by the Lagrange's coef- 
ficients |41j . The extension of this strategy to frequency selective channels 
is given in |42j . 

An explicit characterization of the ergodic capacity region and a simple 
encoding-decoding scheme for a fading GMAC with common data is given 
in |48j . Optimum power allocation schemes are also provided. 



10 Thoughts for practitioners 



Practical schemes for distributed source coding, channel coding and joint 
source-channel coding for MAC are of interest. The achievability proofs 
assume infinite length code words and ignore delay and complexity which 
make them of limited interest in practical scenarios. 

Reference [5] reviews a panorama of practical joint source-channel cod- 
ing methods for single user systems. The techniques given are hierarchical 
protection, channel optimized vector quantizers (COVQ), self organizing 
hypercube (SOH), modulation organized vector quantizer and hierarchichal 
modulation. 

For lossless distributed source coding, Slepian-Wolf (S-W) ([69]) provide 
the rate region. The underlying idea for construction of practical codes for 
this system is to exploit the duality between the source and channel coding. 
The approach is to partition the space of all possible source outcomes into 
disjoint bins that are cosets of a good linear channel code. Such construc- 
tions lead to constructive and non-asymptotic schemes. 

Wyner was the first to suggest such a scheme in [81J. Inspired by Wyner's 
schme, Turbo/ LDPC based practical code design is given in [3] for corre- 
lated binary sources. The correlation between the sources were modelled 
by a 'virtual' binary symmetric channel (BSC) with crossover probability 
p. The performance of this scheme is very close to the Slepian-Wolf limit 
H(p). S-W code designs using powerful turbo and LDPC codes for other 
correlation models and more than two sources is given in [18J . 

LDPC based codes were also proposed in [19] where a general iterative 
S-W decoding algorithm that incorporates the graphical structure of all the 
encoders and operates in a 'Turbo like' fashion is proposed. Reference [51] 
proposes LDPC codes for binary S-W coding problem with Maximum Like- 
lihood(ML) decoding. This gives an upper bound on performance with 
iterative decoding. They also show that a linear code for S-W source coding 
can be used to construct a channel code for a MAC with correlated additive 
white noise. 

In Distributed Source Coding using Syndromes (DISCUS) ([59]) Trellis 
coded Modulation(TCM), Hamming codes and Reed - Solomon (RS) codes 
are used for S-W coding. For the Gaussian version of DISCUS, the source 
is first quantized and then discrete DISCUS is used at both encoder and 
decoder. 

Source coding with fidelity criterion subject to the availability of side 
information is addressed in [82]. First the source is quantized to the extend 
allowed by the fidelity requirement. Then S-W coding is used to remove 



the information at the decoder due to the side information. Since S-W 
coding is based on channel codes, Wyner-Ziv coding can be interpreted as 
a source-channel coding problem. The coding incurres a quantization loss 
due to source coding and binning loss due to channel coding. To achieve 
Wyner-Ziv limit powerful codes need to be employed for both source coding 
and channel coding. 

It was shown in [88] that nested lattice codes can achieve the Wyner-Ziv 
limit asymptotically, for large dimensions. A practical nested lattice code 
implemetation is provided in [67]. For the BSC correlation model, linear 
binary block codes are used for lossy Wyner-Ziv coding in [881 168] . 

Lattice codes and Trellis based codes ([26]) have been used for both 
source and channel coding for the correlated Gaussian sources. A nested 
lattice construction based on similar sublattices for high correlation is pro- 
posed in [67]. Another approach to practical code constructions is based on 
Slepian-Wolf coded nested quantization (SWC-NQ) which is a nested scheme 
followed by binning. Asymptotic performance bounds of SWC-NQ are es- 
tablished in [Hj]. A combination of a scalar quantizer and a powerful S-W 
code is also used for nested Wyner-Ziv coding. Wyner-Ziv coding based on 
TCQ and LDPC are provided in [85] . A comparison of different approaches 
for both Wyner-Ziv coding and classical source coding are provided in [84] . 

Low density generator matrix (LDGM) codes are proposed for joint 
source channel coding of correlated sources in |89j . 

Practical code construction for the special case of the CEO problem are 
provided in [571186]. 

11 Directions for future research 

In this report we have provided sufficient conditions for transmission of 
correlated sources over a MAC with specified distortions. It is of interest 
to find a single letter characterization of the necessary conditions and to 
establish the tightness of the sufficient conditions. It is also of interest to 
extend the above results and coding schemes to sources correlated in time 
and a MAC with memory. The error exponents are also of interest. 

Most of the achievability results in this report use random codes which 
are inefficient because of large codeword length. It is desirable to obtain 
power efficient practical codes for side information aware compression that 
performs very close to the optimal scheme. 

For the fading channels, fairness of the rates provided to different users, 
the delay experienced by the messages of different users and channel tracking 



are issues worth pondering. It is also desirable to find the performance of 
these schemes in terms of scaling behaviour in a network scenario. The 
combination of joint source-channel coding and network coding is also a new 
area of research. Another emerging area is the use of joint source-channel 
codes in MIMO systems and co-operative communication. 

12 Conclusions 

In this report, sufficient conditions are provided for transmission of corre- 
lated sources over a multiple access channel. Various previous results on 
this problem are obtained as special cases. Suitable examples are given to 
emphasis the superiority of joint source-channel coding schemes. Impor- 
tant special cases: Correlated discrete sources over a GMAC and Gaussian 
sources over a GMAC are discussed in more detail. In particular a new 
joint source-channel coding scheme is presented for discrete sources over 
a GMAC. Performance of specific joint source-channel coding schemes for 
Gaussian sources are also compared. Practical schemes like TDMA, FDMA 
and CDMA are brought into this framework. We also consider a MAC with 
feedback and a fading MAC. Various practical schemes motivated by joint 
source-channel coding are also presented. 

A Proof of Theorem 1 

The coding scheme involves distributed quantization (W±, W2) of the sources 
and the side information (Ui, Zi), (U2, Z2) followed by a correlation preserv- 
ing mapping to the channel codewords. The decoding approach involves 
first decoding (Wi, W2) and then obtaining estimate (Ui,U 2 ) as a function 
of (Wi, W2) and the decoder side information Z. We also use the following 
Lemmas in the proof. 

Lemma 6 (^Markov Lemmaj: Suppose X <-> Y <-> Z . If for a given 
(x n ,y n ) £ T™(X,Y), Z n is drawn according to 117=1 P( z i I Vi)> then with high 
probability (x n ,y n ,Z n ) £ T™(X,Y, Z) for n sufficiently large. 

Lemma 7 (Extended Markov Lemmaj: Suppose W\ <-> U\Z\ <-> U2W2Z2Z 

and W 2 U 2 Z 2 UxW^Z. If for a given (uj , u£ , , z% , z n ) G 

T™{U\, U2, Zi, Z2, Z), Wi and W 2 are drawn respectively according to 

Yl2=iP(wii\uu, Zu) and ~[\™' =1 p(w 2 i\u2i, z 2 i), then with high probability 

K, uj, zf, z%, z n , W?, W%) € T e n (C/i, U 2 , Z u Z 2 , Z, W U W 2 ) forn sufficiently 

large. 



The proofs of these lemmas are available in [TO] and [73] respectively. 

We show the achievability of all points in the rate region (1). 

Proof: Fix p(wi\ui,zi),p(w2\u2,Z2),p(xi\wi),p(x2\w2) as well as /$(•) 
satisfying the distortion constraints. 

Codebook Generation: Let i?j = Z^Wi) + 5,i G {1,2} for some 

5 > 0. Generate 2 nR i codewords of length n, sampled iid from the marginal 
distribution p(wi), i G {1,2}. For each wf independently generate sequence 
Xf according to Y\!j=xP{. x ij\ w ii)i * G {1,2}- Call these sequences G 
1,2. Reveal the codebooks to the encoders and the decoder. 

Encoding: For i £ {1, 2}, given the source sequence [/" and Zf, the i*' 1 
encoder looks for a codeword W, n such that (U?,Z?,W?) G T e n ([/i, Z f , Wi) 
and then transmits Xi(W™) where T™(.) is the set of e-weakly typical se- 
quences ([23J) of length n. 

Decoding: Upon receiving Y n , the decoder finds the unique (W^W^) 
pair such that (W?, W?, xi(WT), x 2 (W 2 n ), Y n , Z n ) G T™. If it fails to find 
such a unique pair, the decoder declares an error and incur res a maximum 
distortion of d max . 

In the following we show that the probability of error for the encoding 
decoding scheme tends to zero as n — > oo. The error can occur because of 
the following four events E1-E4. We show that P(Ei) — > 0, for i = 1, 2, 3, 4. 

El The encoders do not find the codewords. However from rate distor- 
tion theory [23], page 356, lim^oo P(E 1 ) = if R ■ > /(£/», Z.; Wi), i G 1, 2. 

E2 The codewords are not jointly typical with Z n . Probobality of this 
event goes to zero from the extended Markov Lemma (Lemma [6]). 

E3 There exists another codeword w™ such that (wf, W% ', xi(wi), ^(W^ 1 ), 

Y n ,Z n ) GT™. Define a= (w?, W?, x 2 {W 2 n ), Y n , Z n ). Then, 

P(E3) = Pr{There is r&f ^ : a € T e n } 

< ^ Pr{aGT™} (58) 

The probability term inside the summation in (j58H is 



< Yl Pr{x 1 (w?),x 2 (w%),y n \w^wlz n } 

(xi(.),x 2 (.),y n ):aeT™ 
(x 1 (.),x 2 (.),y"):a€Tp 

< y] 2- n { H ( Xi \ w ^ +H ( X2 < Y \ W2 ' z *)- 4e } 

(^i(-),^(-),S n ):«eT E " 

< 2 n ^( x i' X 2,^|VKi,H/ 2 ,^)2- n {- H '(^i|M / i)+^(^2,^|H/2,^)-4e} 

But from hypothesis, we have 

H(X 1 ,X 2 ,Y\W 1 ,W 2 ,Z) - fl-(Xi|Wi) - H(X 2 ,Y\W 2 ,Z) 
= H(X 1 \W 1 ) + H(X 2 \W2) + H(Y\X 1 ,X 2 )-H(X 1 \W 1 )-H(X2,Y\W2,Z) 
= H{Y\X 1 ,X 2 )-H(Y\X 2 ,W 2 ,Z) 
= H(Y\X 1 ,X 2 ,W 2 ,Z)-H(Y\X 2 ,W 2 ,Z) 
= -I(Xi;Y\X 2 ,W 2 ,Z). 

Hence, 

Pr{«,VF 2 ",xi«),a; 2 (VF 2 "),y n ,Z n ) G T™} < 2 - n ^ x ^ Y \ x ^ w '^- (ie \ 

(59) 

Then from (1581) 



p(E3) < 2 " rt i / ( Xl ' y l X2 ' H/2 ' Z )" 6e > 

w™ ^w™ : (w™ ,w™ ,z n ) GT™ 
= : K,^,/) G T n|| 2 -n{/(X i; y|X 2 ,W 2 ,Z)-6 e } 

< |{0|Pr«,^,z n ) G Tf}2^ ri ^( Xi;y l X2 ' W2 ' Z )- 6e > 

< 2 n { / ( c/ i' Z i; H/ i)+' 5 }2" ri ^ / ( vyi;W ' 2 ' Z ) _3 ^2 _n{/(Xi;y|X2 ' W2 ' z:) ~ 6 f60) 

_ ^n{I{U^,Zr,Wl\W 2 ,Z)}2~n{I{X 1 ;Y\X 2 ,W2,Z)-<je-5} 

The R.H.S of the above inequality tends to zero if I(U\, Z\\ W\\W 2 , Z) < 
I(X 1 ;Y\X 2 ,W 2 , Z). In ([60]) we have used the fact that 

I^Z^Wt) - I{W±;W 2 ,Z) 

= H(W 1 \W 2 ,Z)-H(W 1 \U l ,Z l ) 

= H(W l \W 2 ,Z)-H(W 1 \U l ,Z l ,W 2 ,Z) 

= I(U U Z X -W X \W 2 ,Z). 



Similarly, by symmetry of the problem we require I(U 2 , Z% W2IW1, Z) < 
I(X 2 ;Y\X 1 ,W 1 ,Z). 

E4 There exist other codewords and w 2 such that 
a=(w^,w^,x 1 (w^),x 2 (w^),y n ,z n ) £ T e n . Then, 

P(E4) = Pr{There is (tof, u>£) / «, u#) : a E T e n } 

(w™ )^(™i : (w? ,w J ,z n ) eT™ 

The probability term inside the summation in (|6ip is 



< X Pr{xi(i&r) J x 2 K),y n |^,i&?,^} 

< V] 2~ n { H ( Xl \ w ^ +H ( X2 \ w ^ +H( - Y \ z '>- 5€ } 

(a;i(.),X2(.),y n ):aGT c " 

< 2 n ^( X i' X 2,^|W / i,VK2,Z) 2 -n{H(X 1 |Vl/i)+//(X2|VK2)+^(y|^)-7e} 

But from hypothesis, we have 

H(X 1 ,X 2 ,Y\W 1 ,W 2 ,Z) - H{X X \W{) - H(X 2 \W 2 ) - H(Y\Z) 

= H(Y\X U X 2 )-H(Y\Z) 

= H(Y\X 1 ,X 2 ,Z)-H(Y\Z) 

= -I{X U X 2 ;Y\Z). 

Hence, 

Pr{{w^w^ 1 x l {w^),x 2 {w r 2 l ),y n ,z n ) £ T™} < 2 -«U(^i.^;Y|Z)-76}_ ^ 
Then from ED) 



P(E4) < X 2 " n{/(Xl ' X2;y|z) - 7e} 

(i£f,i&J,z n )eT™ 

= IUO2) : (w^,w^z n ) E 3^}|2-"W Jf i^: y l z )- 7e > 

< \{wi}\\{w^}\Pr{{w^,w^,z n ) E T ^}2- n ^^ Xl ' X2 ' Y ^- 7 ^ 

< 2 n { / ( l/ i. z i; H/ i)+ / ( c/ 2,^2;W 2 )+2<5} 

2-n{I(Wi;W2,Z)+I(W2;Wi,Z)+I(Wi;W2\Z)-U}2-n{I(X 1 ,X 2 ;Y\Z)-7e} 
_ 2 n { I ( U ^ U 2' Z l^Wi,W2\Z)}2-n{I(X 1 ,X 2 - ) Y\Z)-lle-28} 



The RHS of the above inequality tends to zero if I(Ui , U%, Z\ , Z% W\ Wi \ Z) < 
I{X U X 2 ;Y\Z). 

Thus as n — ► oo, with probability tending to 1, the decoder finds the cor- 
rect sequence (WJ 1 , W 2 ) which is jointly weakly e-typical with (C/f , U 2 , Z n ). 

The fact that (W?,W$) are weakly e-typical with (Uf, U$, Z n ) does 
not guarantee that /jj(Wf , W 2 , Z n ) will satisfy the distortions D\,D 2 . For 
this, one needs that (Wi,W 2 ) are distortion-e-weakly typical ([23]) with 
(f/f, U 2 , Z n ). Let Tg e denote the set of distortion typical sequences ([23]). 
Then by strong law of large numbers P(Tg JT") — > 1 as n — > 00. Thus the 
distortion constraints are also satisfied by (WJ 1 , Wjf) obtained above with 
a probability tending to 1 as n — > 00. Therefore, if distortion measure d is 
bounded lining E[d(U?, U?)] < A + e t = 1, 2. 

If there exist u* such that £?[di(?7i,u*)] < cxd, i = 1,2, then the result 
extends to unbounded distortion measures also as follows. Whenever the 
decoded (W", Wjf) are not in the distortion typical set then we estimate 
({7f,f7 2 n ) as {ul n ,u* 2 n ). Then for i = 1,2, 

17?)] < A + e + ^(^,«r)l{(T« e )=}]- (63) 

Since E[d(J7f , u* n )] < 00 and P[(Tg J c ] -> as n -» 00, the last term of 
(|63j) goes to zero as n — ► 00. 
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